Skip to content

feat: add ASOF join logical semantics - #23829

Draft
Xuanwo wants to merge 12 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-logical
Draft

feat: add ASOF join logical semantics#23829
Xuanwo wants to merge 12 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-logical

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

This is the logical-planning layer of the ASOF JOIN stack. It defines the
logical contract and planner behavior separately from the SQL frontend and
serialization formats.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-physical...xuanwo/asof-logical

What changes are included in this PR?

  • Add LogicalPlan::AsOfJoin, AsOfJoin, and AsOfMatch.
  • Validate deterministic expressions, input ownership, supported match
    operators, equality-key types, and USING constraints.
  • Add LogicalPlanBuilder entry points and schema construction that preserves
    both qualified USING keys while exposing one unqualified wildcard key.
  • Integrate ASOF joins with tree transforms, display, type coercion, projection
    pruning, filter pushdown, and physical planning.
  • Plan logical ASOF joins to the broadcast-based AsOfJoinExec from feat: add ASOF join physical operator #23828.
  • Fail closed at proto, SQL unparser, and Substrait boundaries until their
    owning stack layers add explicit support.
  • Add filter-pushdown and Substrait boundary tests.

Are these changes tested?

Yes:

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion --test core_integration test_asof_join_pushes_only_left_filters --all-features
  • cargo test -p datafusion-substrait asof_join_fails_closed_until_substrait_has_an_extension --all-features

Are there any user-facing changes?

This adds logical-plan and builder APIs for ASOF joins. SQL syntax, DataFrame
APIs, and plan serialization are intentionally left to dependent stack PRs.

As with any new public LogicalPlan variant, downstream exhaustive matches must
add an arm. The variant is appended so existing variants retain their
PartialOrd ordering; maintainers should still treat the enum addition as a
Rust source-compatibility break.

This PR should be reviewed and merged after #23828. After the dependency
squash-merges, this branch will be refreshed from main without rewriting its
published history.

@github-actions github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026
@Xuanwo Xuanwo mentioned this pull request Jul 23, 2026
Draft
@github-actions

github-actions Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details 
     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 109.533s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 104.845s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.599s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 217.417s] datafusion
    Building datafusion-expr v54.1.0 (current)
       Built [  27.042s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.077s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  27.158s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.077s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.225s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  56.567s] datafusion-expr
    Building datafusion-optimizer v54.1.0 (current)
       Built [  27.451s] (current)
     Parsing datafusion-optimizer v54.1.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-optimizer v54.1.0 (baseline)
       Built [  27.576s] (baseline)
     Parsing datafusion-optimizer v54.1.0 (baseline)
      Parsed [   0.030s] (baseline)
    Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.170s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  56.169s] datafusion-optimizer
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  37.009s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.144s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  37.021s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.143s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.623s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  76.127s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  61.107s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  60.403s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.249s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 123.162s] datafusion-proto
    Building datafusion-sql v54.1.0 (current)
       Built [  42.023s] (current)
     Parsing datafusion-sql v54.1.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-sql v54.1.0 (baseline)
       Built [  41.916s] (baseline)
     Parsing datafusion-sql v54.1.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.234s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  85.410s] datafusion-sql
    Building datafusion-substrait v54.1.0 (current)
       Built [ 350.453s] (current)
     Parsing datafusion-substrait v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-substrait v54.1.0 (baseline)
       Built [ 353.119s] (baseline)
     Parsing datafusion-substrait v54.1.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-substrait v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.228s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 705.718s] datafusion-substrait

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026
@codecov-commenter

codecov-commenter commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.33412% with 467 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.64%. Comparing base (e8a65f2) to head (c967002).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-plan/src/joins/asof_join.rs 86.41% 111 Missing and 64 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs 38.70% 100 Missing and 14 partials ⚠️
datafusion/core/src/physical_planner.rs 0.00% 43 Missing ⚠️
datafusion/expr/src/logical_plan/tree_node.rs 0.00% 43 Missing ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 0.00% 28 Missing ⚠️
datafusion/optimizer/src/push_down_filter.rs 0.00% 24 Missing ⚠️
datafusion/expr/src/logical_plan/builder.rs 59.64% 17 Missing and 6 partials ⚠️
datafusion/expr/src/logical_plan/display.rs 0.00% 10 Missing ⚠️
datafusion/optimizer/src/optimizer.rs 0.00% 4 Missing ⚠️
datafusion/proto/src/logical_plan/mod.rs 0.00% 3 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #23829      +/-   ##
==========================================
- Coverage   80.66%   80.64%   -0.03%     
==========================================
  Files        1095     1096       +1     
  Lines      372294   374219    +1925     
  Branches   372294   374219    +1925     
==========================================
+ Hits       300324   301792    +1468     
- Misses      54055    54399     +344     
- Partials    17915    18028     +113

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This was referenced Jul 23, 2026
Draft
Draft
Draft
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

auto detected api change Auto detected API change core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-plan Changes to the physical-plan crate proto Related to proto crate sql SQL Planner substrait Changes to the substrait crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Xuanwo @codecov-commenter